Identification of clusters of companies in stock indices via Potts super-paramagnetic 

transitions 
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The clustering of companies within a specific stock market index is studied by means of super- 
paramagnetic transitions of an appropriate g-state Potts model where the spins correspond to com- 
panies and the interactions are functions of the correlation coefficients determined from the time 
dependence of the companies' individual stock prices. The method is a generalization of the clus- 
tering algorithm by Domany et. al. to the case of anti-ferromagnetic interactions corresponding to 
anti-correlations. For the Dow Jones Industrial Average where no anti-correlations were observed in 
the investigated time period, the previous results obtained by different tools were well reproduced. 
For the Standard & Poor's 500, where anti-correlations occur, repulsion between stocks modify the 
cluster structure. 



Stock market indices, like the Dow Jones (DJ) or Stan- 
dard & Poor's 500 (S&P 500) are used as indicators of 
the status of the markets. They are averaged values of a 
different number of selected companies indicative of the 
economy of a given market. It is of both theoretical and 
practical importance to analyze how individual contribu- 
tions to the average behave. The customary approach in 
the financial literature focuses on the investigation of the 
properties of the covariance matrix. Here we take a dif- 
ferent approach aiming to identify the presence of a hier- 
archical structure inside the set of stocks simultaneously 
traded in a market. The identification of the hierarchy 
of clusters is of central importance both from the point 
of view of understanding the dynamics of the stock index 
and for portfolio optimization (l]J^]. As far as we know 
this question was first analyzed by Mantegna by means 
of the minimal spanning tree method j§-f|, see also 
Here we analyze the problem of clustering of companies 
in the S&P 500 and the DJ indices by a different method 
based on the q-state Potts model which turns out to be 
particularly suitable to handle anti-correlations. 

The idea to use the super-paramagnetic (SPM) order- 
ing of a q-state Potts model for cluster identification is 
due to Domany et. al They start from a set of 

points which lie in a metric space where the mutual dis- 
tances of the points are known. By introducing a dis- 
tance dependent ferromagnetic (FM) interaction between 
Potts spins assigned to the points at appropriately chosen 
temperatures the close points within a cluster feel strong 
interaction and align while far clusters point into differ- 
ent " Potts directions" . The functional dependence of the 
interaction on the distance should be chosen in an appro- 
priate way. For a given interaction the possible hierarchic 
clustering shows up in a series of SPM transitions. 

We have generalized this method by dropping the 



condition of the metric and allowing negative (anti- 
ferromagnetic, AFM) couplings. The coupling between 
the pair of Potts spins {i.e. companies) is in our case the 
explicit function of the correlation coefficient and it is FM 
for positive correlations and AFM for anti-correlations 
(the latter are present in the S&P 500). This way we 
naturally take into account the " repulsion" between neg- 
atively correlated companies (and clusters of companies) 
- an important aspect for portfolio optimization. In order 
to estimate the effect of the anti-correlations we carried 
out calculations with only FM interactions {i.e. we took 
the absolute values of the correlation coefficients) and 
with correctly signed interactions too. We found that 
the difference - in our set of data - can be observed only 
at the ground state, i.e. for the main (dominant) cluster 
structure. 

Consider a q state, inhomogeneous Potts model: Sj = 
1, q, where i = 1, N. N is the number of points one 
should arrange in clusters (the number of companies in 
the considered situation). The cost function will be the 
Hamiltonian: 
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The coupling Jij is a function of the correlation coef- 
ficient dj between the time evolution of the logarithmic 
daily price return Yi — log(Pj(i)) — log(P,(t — 1)) of the 
stock of companies i and j. Pj(t) is the closure price of 
the stock i at the day t. The correlation coefficient can 
be computed as follows: 



G[-l,l]. (2) 



Here (...) is a temporal average performed on all the trad- 
ing days of the investigated time period which ranges 
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from July 3rd, 1989 to October 27th, 1995. 

The Potts model can be used for cluster identification 
in the following way. Let us first consider the simpler, 
FM case 1 , i.e., Jy > 0. These couplings are functions 
of the property the clustering should be based upon - 
in our case this is the correlation coefficient. If the set 
of spins are interrelated in a way that each pair of spins 
can be connected through a path via non- vanishing J^-s 
the ground state of the system is all spins pointing into 
one Potts-direction, i.e., they build a single cluster. As 
the temperature is increased, weak bonds break easier 
than the strong ones and transition to a SPM phase takes 
place where clusters of spins have a specific Potts magne- 
tization but the net magnetization of the whole system 
is zero. The clusters identified in this manner are those 
we have been looking for. Depending on the interactions, 
the system may go through a sequence of such transitions 
signalizing the hierarchical cluster structure. The transi- 
tions are best indicated by monitoring the peak structure 
in the susceptibility and the clusters are then identified 
by means of the spin-spin correlation functions. 

The method is easily generalized to the case where re- 
pulsion between pairs of points is present, in our case, if 
there are anti-correlations between companies as it is the 
case e.g. for the S&P 500. This latter case is important 
when AFM interactions are also present, because of low 
temperature behaviour. 

For the above reasons we make the following choice for 
the interaction: 



Jij = sgn(cy ) 1 - exp 



n — 1 re 



(3) 



where cy is the correlation coefficient between compa- 
nies i and j. The parameters (a, n) should be chosen so 
that the super-paramagnetic state exist, but inside this 
region the result will be not too sensible on the choice. 
The fine tuning serves to be able to observe the transi- 
tions more clearly, i.e. make peaks in the susceptibility 
function sharper, and the constant regions between them 
larger. A possible determination of parameter a is the av- 
erage of the largest correlation coefficients for each spin: 
a = ^/N^2^ =1 maxj (cy). The power n tunes the range 
of interaction, the factor n/ (n — 1) in the exponent shifts 
the inflection point of the interaction function. 
The order parameter is: 



Nmax/N - 1/q 

1-1/5 ' 
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of spins which are in the same state. The value of the 
parameter q is determined by a trial and error optimiza- 
tion: Too small q hinders the identification of the SPM 
clusters since different clusters are then forced to point 
into the same "Potts direction". Too large q makes the 
calculations more cumbersome. The results depend only 
weakly on the value of q. 

The (first order-like) SPM transitions are identified by 
the peaks in the susceptibility \ — N((m 2 ) — (m) 2 )/T, 
which has to be measured as the function of the temper- 
ature. (For more convenience we studied the x(T) = 
Tx(T)/N function.) In the simplest non-trivial case 
there are two transitions as temperature is increased: FM 
-> SPM and SPM -> PM. However, inside the range of 
the SPM phase one can often observe several peaks too, 
meaning that there are more than one characteristic clus- 
ter configurations. As the temperature increases the clus- 
ters break up into sub-clusters: A hierarchical structure 
is revealed. 

The susceptibility x, is approximately constant be- 
tween two peaks. At this temperature regime the clusters 
are identified by means of the spin-spin correlation func- 
tion: If the correlations between two spins exceed a given 
threshold (e.g. 0.5), the spins (companies) are considered 
to belong to the same cluster. The result is not sensitive 
on the choice of the threshold value. The distribution of 
the spin-spin correlation has two peaks, one of them is 
near to the value zero and the other near to the value 
one. The probability that the correlation of two spins 
lies between these two values is low. 

First we analyzed the companies of the Dow Jones in- 
dex which includes N — 30 companies. No negative cor- 
relation coefficient was found for this data set (cy > 0). 
In this purely FM case the simulation could be done with 
the efficient Swendsen-Wang method juj. The parame- 
ters of the interaction (|j|) were set to a = 0.43, n = 8. 
The temperature dependence of the susceptibility x, is 
shown in Fig. |]. 



where q is the number of states that a spin can have, N 
is the number of spins, and N max is the maximal number 
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x It should be mentioned here that there are no anti- 
correlations in the DJ index in the investigated time period. 
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FIG. 1. Temperature dependence of \ = T\/N in the FM case 
(Dow Jones companies). The parameters are: a = 0.43, n = 8 and 
q = 100. 

We analyzed the spin-spin correlations at four different 
temperature levels. We got the following clusters (we use 
the standard notation for the companies): 
T = 0.03 

1) AA, ALD, AXP, BA, BS, CAT, CHV, DD, DIS, GE, GM, 
IP, JPM, KO, MCD, MMM, MO, MRK, PG, T, TX, UK, 
UTX, XON, 
T = 0.13 

1) AA, DD, GE, GM, IP, JPM, KO, MCD, MMM, MO, MRK, 
PG, T 

2) CHV, TX, XON 
T = 0.15 

1) CHV, TX, XON 

2) DD, GE, IP, KO, MCD, MMM, MO, MRK, PG, T 
T = 0.28 

1) CHV, TX, XON 

Due to the small system size we have large fluctua- 
tions and the identification of clusters is not unambigu- 
ous. Nevertheless, the clusters do well match with the 
deterministic minimal spanning tree calculation by Man- 
tegna though the hierarchical structure of the clusters is 
not so clearly represented. 

The other system we analyzed were the companies of 
the S&P500 consisting of N = 443 companies. First we 
looked at the case of strictly FM interactions, i.e. we took 
the absolute value of the correlation coefficients. The 
Fig. |^ presents the x{T) function. The observed large 
fluctuations in the susceptibility function is related to 
the narrow distribution of correlation coefficients (Fig. |^). 
This makes the separation of clusters sensitive to the tem- 
perature. 
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FIG. 2. Temperature dependence of % = Tx/N in the FM case 
(S&P500 companies) with the parameters: a = 0.65, n = 10 and 
q = 50. 

This system represents a clear hierarchical structure 
of the clusters, see Fig. |3|. In the boxes the number of 
the elements of the cluster are indicated. The efficiency 



of the method is represented by the fact that the figure 
matches well with the corresponding result obtained by 
the minimal spanning tree method, including the specific 
composition of the clusters. E.g., at the lowest level of 
the hierarchy (highest temperature in the SPM phase) 
the different industrial branches can be clearly identi- 
fied: Oil, electricity, gold mining, etc. companies build 
separate clusters. 




FIG. 3. The hierarchical structure of clusters of the S&P500 
companies in the FM case. The clusters consisting of single com- 
panies are not indicated. 

It is misleading in the above picture that we take strong 
anti-correlations equal to strong correlations: We have 
to take into account the anti-correlations properly. To 
simulate an inhomogeneous Potts model with long range 
and altering sign interactions seems to be a hard task, 
even the definition of the order parameter is non-trivial. 
Fortunately, in our case the negative interactions play a 
secondary role as compared to the ferromagnetic ones: 
They are much weaker and their number is much less. 
This is shown in Fig. |i| where the distribution of the cor- 
relation coefficients is presented and not the interactions. 
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FIG. 4. Distribution of the measured correlation coefficients 
(S&P500 companies). 

Further simplification is that almost all the negative 
interaction is associated with only five companies (gold 
mining) building one cluster in the FM version too since 
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there are quite strong positive correlations between them. 
The consequence is that except of very low temperatures 
there will be no significant difference to the FM case and 
the simulation of our system can be carried out by a sim- 
ple Metropolis algorithm, there is no need for the applica- 
tion of more sophisticated tools like the multi-canonical 
algorithm. 

However, the determination of the low temperature 
configurations is not straightforward. The system falls 
easily into a local minimum and it takes much simulation 
time to get out of there. Therefore we used a process in 
the spirit of the simulated tempering [jllj] . We excite the 
system to a higher temperature level. Then the temper- 
ature is lowered gradually so that at each temperature 
level we keep the configuration according to the minimal 
energy and keep the records of the best candidates of the 
low energy configurations. 

Fig. |5| represents the configurations and their energy 
values we got for this temperature range. 
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c, 


438,5 


-672.37417 


c 2 


443 


-672.35707 



FIG. 5. Energies and configurations at low temperature in the 
AFM case of the S&P500 companies. 

Clearly, the pure FM state (C2) will not have the low- 
est energy value. This is not very surprising if one knows 
that those five companies have to fall into a separate 
cluster. 

Our goal was to identify the clusters of companies at 
two stock indices, and to show that the repulsion be- 
tween the companies - due to the negative correlation 



coefficients - can modify the cluster structure. Due to 
the distribution of the correlation coefficients in our sys- 
tem this effect is significant only at the ground state, 
i.e., in the dominant cluster structure. Nevertheless, we 
think that our analysis demonstrates the importance of 
the repulsion effects in the clustering problem. 
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